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' Abstract 

o ' 

The simplest explanation for the atmospheric neutrino anomaly is the transition 
of muon neutrinos and antineutrinos into states of different flavor. Data from the 
reactor experiment Chooz, and the measurement of the energy and angle distribu- 
tion of electron-like events in Super-Kamiokande put strong constraints on u^,-v e 
mixing, but the transitions Va — * v T and i/„ — > u s are viable possibilites. If the 
muon neutrinos are mixed with a singlet sterile state, the energy and zenith an- 
gle dependence of the oscillation probability is modified by matter effects, that are 
important for E u /\Am 2 \ > 10 3 GeV/eV 2 . We confront the theoretical predictions 
with existing data on contained events and neutrino-induced upward going muons. 



1 Introduction 

Recent results on atmospheric neutrinos fj show a deficit in the number of ^-induced 
events that can be interpreted as a consequence of neutrino oscillations. Additional signals 
of //-oscillations come from solar neutrinos || and from the LSND experiment [|]]. These 
results, if all interpreted in terms of oscillations, require three independent values of the 
parameter Am 2 , seem to imply the existence of a least four neutrino mass eigenstates || 
and therefore, given the limits on the number of flavors of standard neutrinos jRJ, the 
existence of at least one sterile state. 

The energy and zenith angle distributions of the electron-like events detected by 
Super-Kamiokande are in agreement with the Montecarlo prediction without oscil- 
lations; at the same time, the results of the Chooz experiment M put stringent limits on 
v^-Ve mixing in the region of Am 2 of interest, therefore the hypothesis of z/ M «-> v e oscilla- 
tions as explanation of the atmospheric neutrino data is disfavored (see J] !]] and [E2 for 
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a more quantitative discussion). The atmospheric neutrino data can be explained by the 
existence of <-» v r or <-> za, oscillations, where is a sterile neutrino. The possi- 
bility that the atmospheric neutrino anomaly is due to z^-z/, mixing has been previously 
discussed in fl3|| , and more recenty in |14], ^5|, 



Bounds on the existence of sterile states with large mixing to the standard neutrinos 



have been obtained from cosmological considerations ||18|| . For large enough values of the 
mixing angle and Am 2 , the oscillations of standard into sterile neutrinos can bring the 
sterile states in (or close to) thermal equilibrium with matter before the nucleosynthesis 
epoch. The increase in the energy density results in the overproduction of primordial 
helium, spoiling the success of Big Bang Nucleosynthesis. Schramm and Turner ||19|| , have 
estimated the upper limit (at 95 % C.L.) on the number of light degrees of freedom at 



nucleosynthesis as N v < 3.6. This bound, according to the analyses in [|18[], is incompatible 
with the explanation of the atmospheric neutrino problem as due to the mixing betwen 
muon and sterile neutrinos, because the allowed region in the oscillation parameter space 
would result in N u ~ 4. However recent work pOf has shown that this cosmological bound 
can be evaded, considering the suppression of oscillations due to the possible presence 
of a lepton asymmetry in the early universe that can be generated by the oscillations 
themselves. 

It is interesting to investigate if the data on atmospheric neutrinos already existing or 
to be collected in the near future can discriminate between these two different hypotheses 
for oscillations. The energies of atmospheric neutrinos are such that only a very small 
fraction is above the threshold for r-lepton charged current (CC) production, on the 
other hand the neutral current (NC) cross sections of u T and are identical. Sterile 
neutrinos instead, by definition, do not interact even by neutral currents. In principle 
the presence of a flux of tau neutrinos can therefore be identified as an excess of NC over 
CC events. The measurement is possible in large mass detectors like Super-Kamiokande 
(SK). The most promising method, as recently discussed by Smirnov and Vissani fli"5| , 



is the study of the reaction u x N — » u x Ntc° where a single n° is produced, and detected 
after its decay 7r° — ► 77. The absolute value of the cross section for this process is not 
known with great accuracy, because it has only been measured with large errors PT 



and it is difficult to compute theoretically p2| . Part of this uncertainty should cancel 



15|1 when comparing with the rate of single pion events produced in charged-current 



interactions. Experimentally one will need to know with sufficient accuracy the acceptance 
and efficiency for the detection of events with at least two 'tracks' in the final state. 

Another handle to distinguish the v^—Vr and v^— v s mixing is given by the fact that 
the energy and zenith angle dependence of the oscillation probabilities are different in 
the two cases because of the presence of matter effects f23fl . In this work we discuss 
in detail this possibility, examining the effects that could be observed both in the events 
induced by neutrino interactions in the detector and in the upward going muons, generated 
by neutrino interactions in the rock beneath the detector. As we will show, the most 
promising tool to disentangle matter effects should be provided by the study of upward 
going muons, notwithstanding the present confusing experimental situation. 
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2 Matter effects and oscillation probabilities 



2.1 Oscillation parameters in matter 

The v a — > Up oscillation probability in vacuum, in the simple case of two flavor mixing, is 
given by the standard formula: 

P Va ^ Vp = sin 2 2 9 sin 2 (^^j = sin 2 2 9 sin 2 (V j^j (1) 

where 9, defined in the interval [0, 7r/4], is the mixing angle that relates the flavor and mass 
eigenstates (\v a ) — cos9 \b>i)+sm9 \u2), \vf3) = — sin^ \b>i)+cos9 |z/ 2 )) and Am 2 = ml— ml, 
with mi and m^ the mass eigenvalues. 

In the presence of matter with constant density the mixing angle 9 and the oscillation 
length £ must be replaced by 9 m and £ m , defined as follows: 

2 n sin 2 2 9 

Sm 2 ^ = [C-cos2^ + sin 2 2^ (2) 

D 

(3) 



m {[C-cos2^] 2 + sin 2 2^} 1 /2- 
where 

2_g^ 



and = V a — V/3 is the difference between the effective potentials in matter of neutrinos 
of flavor a and f3 [2~3~|. The effective potentials of ad v T in ordinary matter are identical, 



and matter effects vanish in the case of v tl -v T mixing. In the case of v^-v a mixing the 
difference in the effective potential is 

where the minus (plus) sign is for neutrinos (antineutrinos), Gp is the Fermi constant, 
N n the neutron number density, p the mass density (p D = 5 g cm -3 is a reference value) 
and (1 — Z/A) the neutron fraction. For comparison we note that the effective potential 
in the case of v e -v^ mixing is V eil = ±\^2Gp N e where N e is the electron density; in the 
earth N e ~ N n and V eM ~ -2 V^. 

The meaning of equations (@) and @ is illustrated in figs. [I] and respectively. In 
these figures we have plotted the oscillation parameters in matter for a fixed value of the 
density p = p Q and for values of Am 2 and sin 2 2 9 relevant for atmospheric neutrinos. The 
matter effects are controlled by the ratio £ defined in eq. (^). For |£| <C 1 (small neutrino 
energies) the matter effects are negligible, and the oscillations proceed as in vacuum. For 
|£| ^ 1 (large neutrino energies) the matter effects are dominant and the oscillations are 
strongly supressed: the effective mixing sin 2 2 9 m decreases like (~ 2 and the oscillation 
length levels off to a value £^ = 2tt / |V^ S | ~ 1.3 (p /p) 10 4 km, independent from E v 
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and Am 2 , and remarkably close to the earth's diameter. In the region \(\ ~ 1, that 
corresponds to a neutrino energy 



|Am 2 | „ ( \Am 2 \ \ ( p \ 

E v j= !— = 5.2 GeV ^— — ^ — , 6 

V2G F N n \10- 3 eV 2 J \p) ' 1 ; 

one has the most complex behavior. In general, with the exception of the special case 
of maximal mixing (sin 2 2 9 = 1), neutrinos and antineutrinos have different oscillation 
parameters in matter, because of the opposite sign of the effective potential. The effective 
mixing in matter becomes maximal (MSW resonance effect |2"3fl ) when the condition £ = 
COS26 1 is satisfied. This is possible for antineutrinos (neutrinos) when Am 2 is positive 
(negative) and is illustrated by the curves labeled 'c' and 'd'. 

Inspection of figs. [L] and |2] allows to understand qualitatively when the matter effects 
can be important, and the hypotheses of <-> v T and <-> v s oscillations experimentally 
distinguishable. Atmospheric neutrinos have been observed in essentially three energy 
regions: (a) the sub-GeV region: events with charged lepton energy in the range 0.1- 
0.2 < Ee < 1.33 GeV, produced by neutrinos with average energy (E u ) ~ 0.7-0.8 GeV; (b) 
the multi-GeV region: events with charged lepton energy in the interval E% > 1.33 GeV, 
produced by neutrinos with (E u ) ~ 7 GeV; (c) upward-going muons, produced by a broad 
range of neutrino energies with a median energy of order 100 GeV. The analyses of the 
sub-GeV and multi-GeV data in terms of neutrino oscillations suggest for | Am 2 | a value 
in the range 10~ 3 -j- 10~ 2 eV 2 , therefore matter effects are essentially negligible in the 
sub-GeV region, they can be relevant in the multi-GeV region if |Am 2 | is close to the 
lower end of the above range, and are always significant for upward going muons. 

2.2 Oscillation Probabilities 

In the earth the composition remains always close to the value Z/A ~ 0.5, but the density 
is not constant growing from a value of 2.7 g cm -3 at the surface to 12.5 g cm -3 close to 
the center. Therefore the effective oscillation parameters are not constant, and the flavor 
conversion probabilities must be obtained integrating the flavor evolution equation. 

In fig. ^ we plot the survival probability of muon neutrinos as a function of the zenith 
angle, in the case of maximal mixing with tau, electron and sterile neutrinos. We are 
assuming that the neutrinos are created in a pure state at the surface of the earth, 
and observed when they emerge again from the earth after traveling a distance L = 
—2 R^cosOz (.R© ~ 6371 km is the earth radius). In each panel the curves correspond 
to the survival probability of a neutrino with energy E v = 20, 40, 60 and 80 GeV if 
Am 2 = 5 ■ 10 -3 eV 2 . The oscillation probability is a function of E v j Am 2 and the same 
curves describe the probabilities for different values of Am 2 with a suitable rescaling of 
the values of the energy. 

The upper panel shows the vacuum probabilities (appropriate for v pr v T mixing), that 
are sinusoidal curves of constant amplitude and oscillation length linearly growing with 
E v . In the middle and lower panel the survival probability for <-> v e and <-> v s 
oscillations are plotted. The most remarkable feature to note is the fact that for large E v 
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the curves take an approximately constant shape, with minima at fixed values of the zenith 
angle, but with an amplitude that decreases as E~ 2 . In the case of <-> v e oscillations, 
the asymptotic shape of the transition probability has four maxima at — cos# 2 ~ 0.32, 
0.76, 0.90 and 1. In the case of oscillations into sterile neutrinos the large energy transition 
probability has two maxima, a broad one at cos6 z ~ —0.45 and a sharper and slightly 
higher one at cos^ ~ —0.90 [p4|| . We can note that in the lower panel the curve labeled 
20, that refers to a value E u /Am 2 = 4,000 GeV/eV 2 , is still of an 'intermediate' shape, 
between the vacuum oscillation form and the asymptotic shape of the highest energies; in 
the — > u e case where the matter effects are stronger (y efJL = — 2V^ S ) the corresponding 
curve has a shape closer to the asymptotic one. 

To understand qualitatively these patterns in the oscillation probability it is useful to 
consider briefly propagation in a medium with constant density and composition. In this 
ideal case the oscillation probability can be written as: 

2 r V aP L 



P„ a -+ Vp = sin 2 26 - 6 . 2 sin 2 

p (1 — e cos 26 ) z + e z sin 26 



l-ecos2fl) 2 + e 2 sin 2 26 



, (7) 

with e = ( 1 = Am 2 / (2E l/ V a i3) ■ In the limit of large neutrino energy (that is for e — > 0) 
the transition probability becomes: 



2|2 



I /\ 777 

lim P, ma " er = sin 2 26 ' ' sin 2 

£„->oo v *-* v e ^ElV 2 ^ 



V a pL 



(8) 



that oscillates with an asymptotic oscillation length £™ = 2ir/\V a p\ independent from 
Am 2 and E u , and only a function of the density and composition of the matter, and with 
an amplitude decreasing with the energy as E~ 2 . These are qualitatively exactly the main 
features present in the middle and lower panels of fig. || 

For a fixed value of L, the large energy limit of the oscillation probability in vacuum 
is readily obtained from eq. (JJ) as: 

\Am 2 \ 2 L 2 

lim P v ^ u = sin 2 26 { -— . (9) 

Comparing equations (^) and (^), we note that they differ because in the presence of 
matter the dependence on the distance L has a characteristic oscillating form instead 
of a quadratic increase. In the case of v^—Vg mixing the asymptotic oscillation length 
varies between 23,000 km (at the surface) and 5,300 km (at the earth's center), that 
corresponds to 3.65 and 0.81 earth radii, respectively; for v^-v e) £^ is a factor of two 
shorter. This remarkable coincidence between the oscillation length of high energy v^s 
mixed with electron or sterile neutrinos when traversing the earth and the geometrical 
radius of our planet, generates a characteristic and observable deformation of the zenith 
angle distribution of the high energy neutrinos coming from below the horizon. 

The considerations that we have outlined have been developed for constant density, 
but the qualitative conclusion that for large E v the curve of the oscillation probability as 
a function of zenith angle reaches an asymptotic shape remains valid also in the presence 
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of variable density and composition. In this more general case the shape is not exactly 
sinusoidal; for example one can see from fig. ||| that the maxima of the oscillation proba- 
bility become narrower for neutrino trajectories that pass close to the center of the earth 
where the density is higher and the oscillation length shorter. We note that the density 
of the earth has a sharp discontinuity at a radius r c = 3493 km, the boundary between 
the mantle and the core, where the density jumps from 5.5 to 9.9 g cm -3 ||26|| . A sim- 
ple description of the density of the earth as composed of two layers of constant density 
with a discontinuity at r c , is adequate to describe most of the features of the oscillation 
probabilities, as shown by the dashed curves |27j in the figures. 

In fig. [| we show survival probability curves for the case of a vacuum mixing angle 
with sin 2 29 = 0.7. The upper panel is for <-> u T oscillations, the middle (lower) 
panel describe <-> v s (v^ «-> V s ) oscillations for Am 2 > 0. An MSW resonance is 
present for the antineutrinos when the condition 2 E u (—V fJiS ) / Am 2 = cos 29 is satisfied, 
while the mixing in matter of the neutrinos is always suppressed with respect to the 
vacuum case. In the case Am 2 < one should exchange the oscillation probabilities 
of neutrinos and antineutrinos. We can also note that asymptotically the shape of the 
oscillation probability as a function of zenith angle is the same as in the maximal mixing 
case, however at all energies the oscillation length of neutrinos (antineutrinos) is shorter 
(longer) than the maximal mixing case. The detection of the angular dependence of the 
oscillation probability is therefore more difficult if neutrino and antineutrino events are 
summed together. 



3 Neutrino interactions in the detector 

We discuss first the information that can be obtained from the events where the neutrinos 
interact in the detector via charged currents. In the presence of <-> v T or <-> v s 
oscillations, the rate of muon-like events is reduced, and its zenith angle and energy 
dependence is deformed. The absolute value of the theoretical prediction |29| has a large 



(~ 30%) uncertainty that however to a good approximation affects in equal way the muon 
and electron (anti)neutrino fluxes, and therefore cancels in the analysis of the 'double 
ratio' R = (N^/N e ) data/ (N^/N e ) mc- The Super-Kamiokande collaboration [Q] in an 
exposure of 25.5 kton yrs, has measured R = O.GlO^cis ^ 0-049 for the sub-GeV sample 
and R = 0.659to o5g ± 0.053 for the multi-GeV, showing a significant deviation from the 
expected value of 1. Fitting the energy and zenith angle dependence of the e-like and /z- 
like events (the details are given in |3(], 0]), the SK collaboration has calculated an allowed 
region at 90% C.L. in the (|Am 2 |, sin 2 29) plane for v^-Vt oscillations. It is in principle 
straghtforward to repeat the analysis for the case of z^-za, mixing. Recently Foot, Volkas 
and Yasuda [JT2, [17j have performed precisely this program, using the measurements of R 



and of the angular distributions of /i-like and e-like events to recalculate the allowed region 
for the Vpr v T hypothesis, estimating also the allowed regions for v^— v e and v^— v s mixing. 



The result obtained in [17| is that the regions allowed for u^-Ut and v^-Vg oscillations 



are very similar, with the second one only slightly smaller, because the lowest values of 
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I Am I allowed for oscillations into r— neutrinos are excluded in the sterile case Bill. In 



this work we will not attempt to calculate again an allowed region in parameter space 
for Pfj,-P s mixing. We find that the result of [F7] is qualitatively correct. For a more 
detailed analysis, one woud need to have further informations on the detector acceptance 
and efficiency, and a knowledge of the bidimensional distribution in energy and direction 
of the events, that has not been published. 

We would like instead to argue that the analysis of the zenith angle and energy 
dependence of the suppression of the /i-like events produced in the detector can dis- 
tinguish between the hypotheses of oscillations into tau or sterile neutrinos only for 
| Am 2 1 < 4 • 10 -3 eV 2 . This follows from the facts that: (a) the matter effects, depending 
on the quantity E v pj Am 2 , vanish at low energy; (b) the atmospheric neutrino flux falls 
steeply with energy, and the rate of interactions of high energy neutrinos is small even for 
the large mass of Super-Kamiokande. 

As an illustration in fig. [5] and |^ we show the zenith angle distributions of muons with 
momentum p^ > 0.5 (5.0) GeV produced inside the detector in charged current neutrino 
interactions, and the effects produced by <-> v T and <-> v s oscillations for two 
representative choices of oscillation parameters: maximal mixing and Am 2 = 10~ 3 eV 2 
(fig. |]) or Am 2 = 5 • 10 -3 eV 2 (fig. ||). In order to generate these figures we have used a 
Montecarlo code based on the neutrino flux from Bartol |32| and the neutrino cross section 
as described in ref. ||33|| ; to compute the oscillation probabilities we have also generated 
the neutrino production point according to the analytical approximations described in 
ref. [Q. The event rates calculated (for the position of the Gran Sasso underground 
laboratory) in the absence of oscillations are 82 and 10 events/(kton yr). No effects of 
detector efficiency and resolution have been included. 

In the presence of oscillations the rate of muons events with up-going directions 
(cos #2 < 0) are strongly depleted. For the events with p^ > 0.5 (upper panels) the angle 
between the neutrino and the muon has a broad distribution with average (9^) = 26° 
and this explains the loss of events also for downward-going directions. For the higher 
momentum events (p^ > 5 GeV, lower panels ) the muon direction is more tightly corre- 
lated with the parent neutrino ((9^) = 6.5°), and the depletion of down-going events is 
much smaller. 

The suppression of the rate in the case of v^—Vs oscillations is smaller than in the v^-v T 
case, because the effective mixing is reduced by matter effects. The difference between the 
two cases depends on the muon momentum considered (that is strongly correlated with 
the parent neutrino energy). For low muon momentum the oscillation probabilities of 
the two hypotheses are identical or very similar. With increasing momentum the matter 
effects become progressively more significant reducing the <-» v s oscillation probability. 
This different energy dependence of the oscillation probabilities is in fact the most easily 
detectable signature of the existence of mixing with sterile neutrinos. 

The effect is also strongly dependent on the value of Am 2 . In the case Am 2 = 
10~ 3 eV 2 (fig. |5|) the suppression N^fN of the muon rate with respect to the no-oscillation 
prediction, for the events with p^ > 0.5 GeV is 0.74 (0.78) in the cases of v il -v T (v^-v s ) 
mixing; for the higher momentum cut the suppression becomes 0.83 (0.94) for the two 
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cases, with a much larger difference. The angular dependence of the suppression is also 
different for oscillations into tau or sterile neutrinos |f2Sj| . 

For the higher value Am 2 = 5 ■ 10~ 3 eV 2 (fig. ^[) the suppression is nearly identical 
(0.689 and 0.695) in the case of the lower momentum muons and also for the higher 
momentum sample the difference between the two cases with suppressions of 0.74 and 
0.78 is small. 

To summarize this discussion in fig. [7| we plot the ratio N^/N° of the muon rate to 
the prediction in the absence of oscillations, as a function of Am 2 for the two different 
cuts in the muon momentum, p^ > 0.5 GeV or p^ > 5.0 GeV, both for maximal mix- 
ing (upper half of the figure) and for sin 2 2 9 = 0.80 (lower half). The predictions for 

«-> v T oscillations are given by the full lines, those for oscillations into sterile neutrinos 
by dashed (dot-dashed) lines assuming Am 2 > (Am 2 < 0). For very small |Am 2 | the 
oscillation cannot develop and the ratio A^/iV° ~ 1; with increasing |Am 2 | the ratio de- 
creases monotonously, reaching asymptotically a value 1 — | sin 2 29, corresponding to the 
averaging over rapid oscillations. Note that the region |Am 2 | ~ 2-4- 10 -3 eV 2 corresponds 
to a situation where most up-going (down-going) neutrinos do (do not) oscillate. From 
fig. [7] we can see that for sufficiently large |Am 2 | the curves corresponding to the v^- 
v T and Vp—Us mixing become undistinguishable. Selecting higher momentum muons the 
difference between the two oscillation hypotheses remains significant until larger values 
of | Am 2 1, however the event sample available to study the effects becomes progressively 
smaller. 

A larger number of high energy neutrino events is obtainable from the upward-going 
muon sample. In the remaining sections we discuss this class of events. 



4 Upward— going muons 

Muon neutrinos and antineutrinos can be detected indirectly observing the muons pro- 
duced in CC interactions in the vicinity of the detector. In the presence of — > v x 
oscillations the flux of //-induced muons is suppressed, the energy spectrum is deformed 
and the angular distribution distorted. In essentially the entire region of //-oscillations 
parameter space that is a solution for the atmospheric neutrino problem, the muon flux is 
modified in ways that are in principle observable with large area detectors. The neutrino 
energies that contribute to the muon flux extend up to ~ 1000 GeV, therefore the matter 
effects are important and allow in principle to distinguish between the v pr v T and u^-Ug 
hypotheses. 

The //-induced muon flux is measurable only in directions from below the horizon 



3q1 , therefore it is not possible to measure deviations from the up-down symmetry of 
the no-oscillation flux. The effects of oscillations can be studied only comparing the data 
with a Montecarlo calculation, based on assumptions about the primary cosmic ray flux, 
the properties of particle production in hadronic interactions with air-nuclei, the charged 
current neutrino cross section, and the properties of muon propagation. The theoretical 
uncertainties in the different elements of the calculation result in a sytematic uncertainty 
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of ~ 20% for the absolute normalization of the flux. The energy and especially the zenith 
angle dependence of the flux can be calculated more reliably |36| and are useful probes of 
the existence of oscillations. 

As an illustration of the effect of neutrino oscillations, in fig. || we show the flux of 
upward going muons above a minimum energy E min = 1 GeV plotted as a function of 
the zenith angle. The flux is calculated in the absence of oscillations, and for v^—v^- and 
v^-Vs oscillation parameters corresponding to the SK 'test point' (Am 2 = 5 • 10~ 3 eV 2 
and sin 2 29 = 1). As a best estimate of the muon flux we have used the neutrino flux 
of Bartol H, the cross sections described in |33| and the muon energy loss of |37|]. The 
no-oscillation distribution has a maximum (minimum) for horizontal (vertical) muons, 
that reflects the larger flux of neutrinos for inclined directions. In reference we have 
discussed in some detail the prediction of the upgoing muons zenith angle distribution and 
its theoretical uncertainties. Here we will concentrate on the shape of the distortion of 
the distribution produced by oscillations. In fig. [| we have plotted the ratio between the 
oscillated and non-oscillated flux for different values of Am 2 , always assuming maximal 
mixing. In the case of <-» v T oscillations the distortion factor varies smoothly and 
monotonously from a value close to unity for horizontal muons, to a maximum suppression 
for vertical muons. This behaviour can be easily understood: the neutrinos that generate 
the upward-going muons have a broad energy spectrum, with a shape that (in the absence 
of oscillations) changes only moderately for different zenith angles. For a fixed value of the 
zenith angle, the transition probability Piy^ — > v T ) is a function of the neutrino energy, 
for low E u it oscillates rapidly with an average value ~ sin 2 29, for large E u it decreases as 
L 2 /E 2 , and is negligibly small for E v ^s> L |Am 2 | ~ 2R (S | cos# z | |Am 2 |. With increasing 
| cos 9 Z | neutrinos of higher energy have a chance to oscillate and the muon flux is more 
suppressed. 

In the case of oscillations with maximal mixing into sterile neutrinos, the suppression 
of the flux is smaller than in the corresponding vacuum case because the effective mixing 
in matter is reduced. The distortion factor does not depend monotonously on the zenith 
angle, but presents a structure with two minima at cos9 z ~ —0.35 and —0.9. The origin of 
this structure can be understood as a simple consequence of the considerations developed 
in the discussion of oscillation probabilities. The transition probability Piv^ — > u s ) as a 
function of energy for a fixed zenith angle, is similar to the vacuum case for low E u . For 
larger E v it assumes the form discussed in section 2.2, developing a characteristic shape 
(see the lower panel in fig. ^) that after integration over all neutrino energies remains 
visible in fig. [|. 

The detection of the structure in the distortion factor, present at cos9 z ~ —0.9 would 
be a clear and unambiguous experimental proof of the existence of mixing between muon 
and sterile neutrinos, but requires a large number of events. Easier to detect is perhaps 
the general trend of a suppression with respect to the no-oscillation expectation that 
becomes weaker going from cos9 z ~ —0.3 to cos9 z ~ —0.8, in constrast with the vacuum 
case. 

The qualitative features of the distortion factors described above are present also 
selecting other energy intervals for the uwpard going muons. In fig. 110 we show the 
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distortion factor of the zenith angle distribution of upgoing muons in the interval 1 < 
< 3 GeV selected because it is in principle experimentally accessible, the muon energy 
being estimated from the measured range for stopping particles. The interval of parent 
neutrino energies that contributes to this flux is narrower, extending up to a neutrino 
energy of ~ 50 GeV. This can result in strong features in the angular distribution for 
Am 2 smaller that ~ 10~ 3 eV 2 ; on the other hand for Am 2 approaching 10 -2 eV 2 the 
distortion factor rapidly reaches the constant value 1 — | sin 2 29 both for vacuum and for 
matter oscillations. 

In fig. |ll| we show the distortion factors of the flux of muons with > 1 GeV for 
Am 2 = 5 • 10~ 3 eV 2 and sin 2 29 = 0.7. In the case of vacuum oscillations the distortion 
factor for sin 2 29 < 1 can be obtained from the result for maximal mixing simply as: 

(P(Am 2 , sin 2 29)) = sin 2 29 (P(Am 2 , 1)) (10) 

where (P) = 1 — M / 0° is the fractional difference between the non oscillated and oscil- 
lated flux. In the presence of matter effects such a simple relation does not exist, and 
the dependence on the mixing angle in non trivial. The structure in the distortion factor 
is in general less visible in the case of non-maximal mixing. This is because the oscilla- 
tion lengths of neutrinos and antineutrinos are different in the crucial region around the 
MSW resonance (see the curves labeled 'c' and 'd' in fig. |2|) and this results in different 
angular patterns of the distortion for neutrinos and antineutrinos, as shown in fig. |j. It 
is important also to note that the sign of the mass difference, non observable in vacuum, 
is important in this case. 



5 Upward— going muon data 

Several detectors have collected data on neutrino-induced muons: Baksan 1051, MACRO 



3^], Kamiokande [0, 1MB |]41 |, and Super-Kamiokande @. The situation at present 
is somewhat confused because of the poor agreement between different experiments, and 
between the data and theoretical expectations even after the inclusion of oscillations. 

To compare theoretical predictions with experimental data we assume that the the- 
oretical estimate of the flux is affected by systematic uncertainties that to a good ap- 
proximation enter as a constant normalization factor without affecting the shape of the 
distribution p|. We therefore define a x 2 as: 




X 2 = min a V r-L U + __ (H) 



In eq. (fTT|), the sum is over the number of bins in zenith angle (all experiments have been 
using 10 bins of equal width Acos^ = 0.1), <pj and A<pj are the measured value of the 
flux in the bin and its error, (f/- 1 is the calculated value and a is a normalization factor and 
Aa is the theoretical error, estimated as ~ 0.2. The combination of experimental errors 
(A0j's) and theoretical uncertainty (Aa) in the estimate of the x 2 i n ec L- (HP cannot be 
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rigorously justified, but at least qualitatively has the desired features. Note that the limit 
Act — > forces a to be 1, and corresponds to neglecting the theoretical uncertainties; 
the opposite limit Aa — > oo takes into consideration only the shape of the zenith angle 
distribution. 



5.1 Super— Kamiokande 

The Super-Kamiokande collaboration ||] has presented data on through-going muons, 
that is tracks that enter in and exit from the detector, with a minimum path-length 
L m in = 7 m. To compare with the experimental data we have calculated a muon flux as: 

Acosfl 2 ) = — - 7TT / dE tl -^(E fl ,cos9 z )[A(L min ),9 z )-A(L(E fl ),9 z )} (12) 

where cf</>* / dE^ is the flux of neutrino induced muons with energy and zenith angle 9 Z , 
L(E) and E(L) are the muon range in water and its inverse function, and A(L, 9 Z ) is the 
projected area of the detector that corresponds to trajectories with internal pathlength 
longer than L. The factor [A(L min , 6 Z ) — A(L(E^), 9 Z )] weights the contribution of muons 
of energy E^ to the flux, taking into account the fact that muons of low energy stopping in 
the detector should not be counted. Assuming for Super-Kamiokande the ideal geometry 
of a cylinder (with radius R = 16.9 m and height H = 36.2 m), a simple geometrical 
calculation gives: 

A(L, 9 Z ) = 2 RH sin 9 Z Vl - x 2 + 2 R 2 \ cos9 z \ [cos" 1 x - 3 x VI - x 2 ] Q[L max (9 z ) — L) , 

(13) 

with x = Lsm9 z /2R and L m3iX (9 z ) = min[2i?/ sin# 2 , H/\cos9 z \}. 

Our calculation of the //-induced muon flux is in excellent agreement with the one 
presented by the SK collaboration f2|, indicating that our simple geometrical picture of 
the acceptance is a reasonable approximation. The main difference between our results 
and the theoretical curves presented by SK is a higher (~ 5%) absolute normalization of 
our estimate, probably due to the fact that we model the neutrino cross section including 
an explicit treatment of the quasi-elastic scattering and single-pion production p3 |. 



An illustration of the effect of oscillations is shown in fig. [12], where we compare the 
SK data points with (i) a no-oscillation prediction (full line), (ii) <-> v T oscillations 
with sin 2 2$ = 1 and Am 2 = 10~ 2 eV 2 (dashed line), (iii) *-> v s oscillations with the 
same parameters (dot-dashed line). The three models have been calculated with the same 



theoretical input (4> u from Honda et al. |42 ], a u as described in with leading order 



PDF's from GRV (43f), and rescaled with factors a = 0.88, 1.21 and 1.09. All three 
models give acceptable fits to the data, with \ 2 values of 12.8, 10.0 and 6.2. We can 
observe: (i) the no-oscillation hypothesis tends to be too flat, missing the data point for 
near-to-horizontal muons; (ii) the hypothesis of <-> v T oscillations with maximal mixing 
gives a high ratio between the horizontal and vertical flux, and in this hypothesis the best 
fit (x 2 ~ 8) is obtained for sin 2 29 ~ 0.70; (iii) the hypothesis of <->• v s oscillations 
with maximal mixing gives a suprisingly good fit to the data. We would like to stress 
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the importance of the measurement of near-to-horizontal muons (—0.1 < cos# z < 0) in 
determining the results of the fits. 

For the no-oscillation hypothesis, the x 2 = 12.8 has been computed assuming Aa = 
0.2. Assuming instead Aa = the x 2 takes the value 20.4 (probability 2.6 ■ 10~ 2 ). 
Neglecting the absolute normalization (assuming Aa — ► oo) the optimum a becomes 0.87 
with x 2 decreasing to 12.4; these numbers can be compared with the estimates: a = 0.83 
and x 2 — 12.7 of the SK collaboration || done with the same neutrino flux [42]|. Using the 
Bartol neutrino flux [32] we find a result that is 6.5% higher in absolute normalization with 
small differences in the angular distribution: for Aa = 0.2 the x 2 f° r the no-oscillation 
hypothesis is 15.1. 

The x 2 resulting from fits to the data using the oscillation hypotheses (either <-> v T 
or <-> u s ) are presented in fig. [13| as a function of Am 2 for sin 2 29 = 1 and 0.8. It is 
interesting to note that the inclusion of neutrino oscillations, with parameters in the range 
suggested by the analysis of (semi-) contained events, does result in a better agreeement 
between data and prediction. The best fit (x 2 — 6.1) is obtained for <-> v s oscillations 
with Am 2 ~ 1.4- 10~ 2 eV 2 and sin 2 2 6 ~ 1 (with a ~ 1.12). Assuming *-> v T oscillations 
the x 2 is minimized (x 2 — 7.8 ) for Am 2 ~ 2.2- 10~ 2 eV 2 and sin 2 26 ~ 0.7 (with a = 1.18). 

Since our estimate of the theoretical error Aa ~ 0.2 is rather arbitrary, we have 
repeated the fits for other choices of Aa. The — > v s hypothesis has always a lower 
X 2 , because it predicts a shape that is in closer agreement with the data and requires a 
normalization factor a closer to unity. Large values of Am 2 require a normalization factor 
considerably larger than unity, therefore an increased weight of the absolute normalization 
(i.e. smaller values of Aa) lowers the best-fit values of Am 2 . The limiting case of Aa = 
is illustrated in fig. [14|. It may be noted that in this case, the range of parameters that 
minimize the x 2 f° r oscillations into sterile neutrinos is very similar to the one that fits 
the sub-GeV and multi-GeV SK data according to Foot et al. , as one can see comparing 
fig. H with fig. 12 in |L7 ] . 



In summary, the SK data on upward going muons are consistent with the existence 
of neutrino oscillations, even if still inconclusive if taken by themselves. Considering the 
reduction of statistical errors expected with a longer exposure, future data should allow 
to distinguish the hypothesis of v^-v T and v^-Vs mixing. 

Additional information on the existence of neutrino oscillations can be obtained from 



the analysis of neutrino induced muons that stop in the detector |41j (see also [JT3|, |4|, |36 
for additional discussion). In fig. |T5] we present predictions of the absolute rates and 
angular distributions of upgoing muons stopping in Super-Kamiokande. As discussed in 
previous works, in the presence of oscillations the flux of stopping muons is suppressed 
more strongly than the flux of through-going particles. The typical neutrino energies that 
produce stopping muons are similar to those that are the origin of multi-GeV events, and 
therefore matter effects have approximately the same importance. The suppression of 
the stopping flux and the distortion of the angular distribution depend on the type of 
oscillations only for small |Am 2 |. 
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5.2 Baksan and MACRO. 



Data samples on upgoing muons with comparable statistics have also been collected by 
other experiments. These data are difficult to describe both with and without oscillations, 
and also in poor agreement with each other j|6]. . 

The Baksan and MACRO experiment have presented their results giving the muon 
flux above a threshold energy of 1 GeV correcting for the detector acceptance. We note 
that, as remarked in fl36"l , the detectors do not have a sharp, angle independent threshold, 
and therefore the corrections are in principle model dependent. 

We present in fig. [16] the data of the Baksan and MACRO detectors, together with 
fitted theoretical predictions. The fits are quite poor. 

For Baksan the x 2 °f the no-oscillation hypothesis is x 2 — 26.6 with a corresponding 
normalization factor a ~ 0.95. Including <-» v T [y^ <-> u s ) oscillations does not result 
in a significantly better fit to the data: the minimum x 2 is 25.7 (23.2) for Am 2 ~ 1.0 ■ 
lO" 3 eV 2 , sin 2 2^ ~ 0.70 and a = 1.17 (Am 2 ~ -1.1 • 10~ 3 eV 2 , sin 2 2^ ~ 0.90, a = 1.08). 

For MACRO the \ 2 f° r the no oscillation hypothesis (considering only statistical er- 
rors) is x 2 — 39 (with a ~ 0.70). Including <->• v T {y^ <-> v s ) oscillations the x 2 is 
minimized (x 2 = 30.2 and 35.5) with the choice of parameters: Am 2 ~ 1.5 • 10~ 3 eV 2 , 
sin 2 26 ~ 1, a = 0.97 (Am 2 ~ 1.5 • 10" 2 eV 2 , sin 2 26 ~ 1, a = 1.02). In the <-> v T 
hypothesis the x 2 is reduced by 7 units, however given its large value the significance of 
the result is not clear. 

The zenith angle distribution of the MACRO data shows some unexplained structure 
with a deficit for directions close to the vertical and an excess for cos^ ~ —(0.6-0.7). Liu 
and Smirnov Jl6| have suggested that this structure could be evidence for oscillations of 
muon into sterile neutrinos, observing that the qualitative features of the measured angu- 
lar distribution are similar to those of the asymptotic form of the oscillation probability 
(see fig. 3). A more detailed calculation does not support this claim. In the case of v^-Vs 
mixing the matter effects do predict features like a reduction of the flux for cos^ ~ —0.9, 
however the amplitude of the effect is much smaller than what is measured by MACRO. 



6 Conclusions 

The simplest explanation for the atmospheric neutrino anomaly is the transition of muon 
neutrinos and antineutrinos into tau or sterile (anti)neutrinos. If the muon neutrinos 
are mixed with a singlet sterile state, the energy and zenith angle dependence of the 
oscillation probability is modified by matter effects. The differences between the cases of 
v^-Vr and v^—Vs mixing are important for E u /\Am 2 \ > 10 3 GeV/eV 2 . The two cases are 
distinguishable with high statistics studies of multi-GeV events, but only for the lower 
range of values of |Am 2 | allowed by the analysis of (semi-) contained events. For larger 
| Am 2 1, the distinction will be possible through a detailed study of upgoing muons. The 
muon data available at present are of difficult interpretation, in that only the data obtained 
by the Super-Kamiokande collaboration are in agreement with theoretical expectations. 
Moreover, the Baksan and MACRO experimental data disagree with each other. The 



13 



SK data favour neutrino oscillations, albeit with low statistical significance; both <-> v s 
(with slightly better agreement) and <->• u T oscillations provide good fits. We can expect 
that forthcoming additional data will clarify the situation and allow to identify the right 
solution. 
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Figure 1: Effective mixing parameter for v^-Vg oscillations in matter with constant density 
p = 5 g cm" 3 and neutron fraction 0.5. 
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Figure 2: Oscillation length in matter. 
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Figure 3: Survival probability Piy^ — > v^) as a function of the zenith angle in the case of 
maximal mixing of with v T (upper panel), v e (middle panel) and v s (lower panel). For 
| Am 2 1 = 5 • 10~ 3 eV 2 the curves correspond to neutrino energies 20, 40, 60 and 80 GeV. 
The dashed curves are calculated with the approximation of constant average densities in 



the mantle and in the core of the earth [27 
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Figure 5: Upper panel: Angular distribution of the muon events with > 0.5 GeV, in 
the absence of oscillations (histogram a), and in the presence of oscillations with Am 2 = 
1CT 3 eV 2 and maximal v^-v? (b) or v^-v s (c) mixing. Lower panel: the same, with 
Vil > 5.0 GeV. 
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Figure 7: Ratio of the rate of muon events to the no-oscillation case as a function of Am 2 , 
for two choices of the minimum muon momentum. The upper (lower) panel is calculated 
for sin 2 29 = 1 (0.8). The solid (dashed) curves are for u^-u T (v^-Vs) mixing. In the lower 
panel the dot-dashed curve describes v^-Vg mixing with Am 2 < 0. 
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Figure 9: Ratio between the upward-going muon flux (E^ > 1 GeV), and the no- 
oscillation prediction for maximal mixing and (starting from the highest line) Am 2 = 
1(T 3 - 5 , 10~ 3 , 1(T 2 - 5 and 10~ 2 eV 2 . The solid (dashed) lines are for v^-Vt (y^-Vs) mixing. 
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Figure 10: As in fig. || with the upward going muon flux calculated in the energy interval 
1< E a < 3 GeV. 
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Figure 12: Flux of through-going upward going muons measured by Super-Kamiokande 
[H, compared with theoretical predictions with and without oscillations (see text) 
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Figure 13: x 2 of fits to the upward-going muons data of Super-Kamiokande calculated 
according to eq. 11 plotted as a function of Am 2 , for sin 2 29 = 1 and 0.8 assuming Vn-v T 



and z/^-z/ s mixing. The uncertainty Aa on the absolute normalization is 0.2. 
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Figure 15: Predictions of the rate and angular distribution of stopping muons in SK, in 
the absence (and presence) of oscillations. The subscript r (s) indicates v^-Vr {y^-Vs) 
mixing. 
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Figure 16: Flux of through-going upward going muons measured in the Baksan P5 
(upper panel) and MACRO (lower panel) detectors, compared to the no-oscillation 
hypothesis and to oscillation models for v^—v^- and v^— v s with best fit parameters (see 
text). 
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